perf(hadoop): Use precompiled/hadoop for faster image builds#1472
Open
NickLarsenNZ wants to merge 5 commits intomainfrom
Open
perf(hadoop): Use precompiled/hadoop for faster image builds#1472NickLarsenNZ wants to merge 5 commits intomainfrom
precompiled/hadoop for faster image builds#1472NickLarsenNZ wants to merge 5 commits intomainfrom
Conversation
NickLarsenNZ
force-pushed
the
chore/faster-hadoop-builds
branch
from
e089749 to
3caea9c
Compare
Member
Author
|
Waiting on #1474 |
Note: this commit is being used to illustrate an issue where the timestamp appears as the version in the HDFS web UI instead of the SDP release
Member
Author
|
Caution BLOCKERWe've run into an issue. It is mostly a cosmetic thing (albeit important to customers), but we also need to consider what this means in terms of SBOMs and lineage. Note I think this didn't appear on my initial PoC because I happened to use As part of the faster builds work, we compile a version+patchset of some code (eg: what was This requires some changes (see: 0e79589), which leads to the timestamped version in the HDFS web UI. 
@lfrancke found where in the code the web UI gets the version string from. # Run as root so we can install unzip
docker run --rm -it --user 0 oci.stackable.tech/precompiled/hadoop:3.4.2-stackable1776755865
microdnf update && microdnf install unzip
cd /stackable/hadoop-3.4.2-stackable1776755865/share/hadoop/common
unzip hadoop-common-3.4.2-stackable1776755865.jar
cat common-version-info.properties
# ...
version=3.4.2-stackable1776755865
revision=08a7206a29f212e1f0e3bd81e0cb0be7907907a4
branch=patchable/3.4.2
user=stackable
date=2026-04-21T07:38Z
url=Unknown
srcChecksum=e573e429c0e2b636ead65e1a9b8bd46a
protocVersion=3.23.4
compilePlatform=linux-x86_64We could add a light build stage to the product image that extracts the file, updates the version string, and archives it again. 
This needs further discussion with @lfrancke, @StefanFl, @dervoeti. |
Member
Author
|
Just noting that tests pass at 0e79589 Detailscargo boil build hadoop=3.4.2 --strip-architecture
# Yes, I forgot to tell boil to do this
docker tag oci.stackable.tech/sdp/hadoop:3.4.2-stackable0.0.0-dev \
localboi/hadoop:3.4.2-stackable0.0.0-dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1465
The precompiled/hadoop was done in #1466